Method and score management node for supporting service evaluation

ABSTRACT

A score management node receives network measurements related to at least one service event when the service is delivered to the user, and determines, in a first scoring module, a quality score Q reflecting the user&#39;s perception of quality of the delivered service and an associated significance S reflecting the user&#39;s perception of importance of the delivered service, based on the received network measurements. The determined quality score Q and associated significance S of each service event are modified in a succession of intermediate scoring modules, based on a predefined influence factor applied in each intermediate scoring module. The perception score P is further calculated in a concluding scoring module, based on the modified quality score and associated modified significance, wherein the calculated perception score P is made available for use in the service evaluation.

TECHNICAL FIELD

The present disclosure relates generally to a method and a scoremanagement node for supporting service evaluation by obtaining aperception score P reflecting a user's experience of a service deliveredby means of a telecommunication network.

BACKGROUND

When a service has been delivered by means of a telecommunicationnetwork by a service provider to one or more users, it is of interestfor the service provider to know whether the user is satisfied with thedelivered service or not, e.g. to find out if the service hasshortcomings that need to be improved in some way to make it moreattractive to this user and to other users. Service providers, e.g.network operators, are naturally interested in making their services asattractive as possible to users in order to increase sales, and aservice may therefore be designed and developed so as to meet the users'demands and expectations as far as possible. It is therefore useful togain knowledge about the users' opinion after service delivery in orderto evaluate the service. The services discussed in this disclosure may,without limitation, be related to streaming of audio and visual contente.g. music and video, on-line games, web browsing, file downloads, voiceand video calls, delivery of information e.g. in the form of files,images and notifications, and so forth, i.e. any service that can bedelivered by means of a telecommunication network.

A normal way to obtain the users' opinion about a delivered service isto explicitly ask the customer, after delivery, to answer certainquestions about the service in a survey or the like. For example, theservice provider may send out or otherwise present an inquiry form,questionnaire or opinion poll to the customer with various questionsrelated to user satisfaction of the service and its delivery. If severalusers respond to such a poll or questionnaire, the results can be usedfor evaluating the service, e.g. for finding improvements to make,provided that the responses are honest and that a significant number ofusers have answered. An example of using survey results for estimatingthe opinion of users is the so-called Net Promoter Score, NPS, which iscalculated from answers to user surveys to indicate the users' collectedopinions expressed in the survey answers.

However, it is often difficult to motivate a user to take the time andtrouble to actually answer the questions and send a response back to theservice provider. Users are often notoriously reluctant to provide theiropinions on such matters, particularly in view of the vast amounts ofinformation and questionnaires flooding users in the current modernsociety. One way to motivate the user is to reward him/her in some waywhen submitting a response, e.g. by giving some present or a discounteither on the purchased services or when buying future services, and soforth.

Even so, it is a problem that surveys can in practice only be conductedfor a limited number of users which may not be representative for allusers of a service, and that the feedback cannot be obtained in“real-time”, that is immediately after service delivery. A survey shouldnot be sent to a user too frequently either. The obtained feedback maythus get out-of-date.

Further problems include that considerable efforts must be spent todistribute a survey to a significant but still limited number of usersand to review and evaluate all answers coming in, sometimes with poorresults due to low responsiveness. Furthermore, the user may provideopinions which are not really accurate or honest and responses tosurveys may even be misleading. For example, the user is often prone toforget how the service was actually perceived or experienced when it wasdelivered, even after a short while, once prompted to respond to aquestionnaire. Human memory thus tends to change over time, and theresponse given may not necessarily reflect what the user really felt andthought at service delivery. The user may further provide the responsevery hastily and as simply as possible not caring much if it reallyreflects their true opinion. The opinion expressed may also be dependenton the user's current mood such that different opinions may be expressedat different occasions, making the response all the more erratic andunreliable.

Still another problem is that it can be quite difficult to trace anunderlying reason why users have been dissatisfied with a particularservice, so as to take actions to eliminate the fault and improve theservice and/or the network used for its delivery. Tracing the reason forsuch dissatisfaction may require that any negative opinions given byusers need to be correlated with certain operational specifics relatedto network performance, e.g. relating to where, when and how the servicewas delivered to these users. This kind of information is not generallyavailable and analysis of the network performance must be done manuallyby looking into usage history and history of network issues. Muchefforts and costs are thus required to enable tracing of such faults andshortcomings.

SUMMARY

It is an object of embodiments described herein to address at least someof the problems and issues outlined above. It is possible to achievethis object and others by using a method and a score management node asdefined in the attached independent claims.

According to one aspect, a method is performed by a score managementnode for supporting service evaluation by obtaining a perception score Preflecting a user's experience of a service delivered by means of atelecommunication network. In this method, the score management nodereceives network measurements related to at least one service event whenthe service is delivered to the user. The score management nodecomprises functional scoring modules which are used as follows.

The score management node determines, in a first scoring module, aquality score Q reflecting the user's perception of quality of thedelivered service and an associated significance S reflecting the user'sperception of importance of the delivered service, based on the receivednetwork measurements. The score management node then modifies, in asuccession of intermediate scoring modules, the determined quality scoreQ and associated significance S of each service event based on apredefined influence factor applied in each intermediate scoring module.The score management node then calculates, in a concluding scoringmodule, the perception score P based on the modified quality score Qmand associated modified significance Sm, wherein the calculatedperception score P is made available for use in the service evaluation.

According to another aspect, a score management node is arranged tosupport service evaluation by obtaining a perception score P reflectinga user's experience of a service delivered by means of atelecommunication network. the score management node comprises aprocessor and a memory containing instructions executable by theprocessor, whereby the score management node is configured to:

-   -   receive network measurements related to at least one service        event when the service is delivered to the user,    -   determine, in a first scoring module, a quality score Q        reflecting the user's perception of quality of the delivered        service and an associated significance S reflecting the user's        perception of importance of the delivered service, based on the        received network measurements,    -   modify, in a succession of intermediate scoring modules, the        determined quality score Q and associated significance S of each        service event based on a predefined influence factor applied in        each intermediate scoring module, and    -   calculate, in a concluding scoring module, the perception score        P based on the modified quality score Qm and associated modified        significance Sm, wherein the calculated perception score P is        made available for use in the service evaluation.

Thereby, the perception score P can be used in the service evaluation asan estimation of the users' opinion and it is possible to obtain Pautomatically after every time a service is delivered to the user.Further, the perception score P is calculated from technicalmeasurements in the network related to the service usage which arereadily available for any user and it is thus not necessary to depend onthe user to answer a survey or the like.

The above method and score management node may be configured andimplemented according to different optional embodiments to accomplishfurther features and benefits, to be described below.

A computer program storage product is also provided comprisinginstructions which, when executed on at least one processor in the scoremanagement node, cause the at least one processor to carry out themethod described above for the score management node.

BRIEF DESCRIPTION OF DRAWINGS

The solution will now be described in more detail by means of exemplaryembodiments and with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram illustrating an example of how a scoremanagement node may be configured and operate, according to somepossible embodiments.

FIG. 2 is a flow chart illustrating a procedure in a score managementnode, according to further possible embodiments.

FIG. 3 is a block diagram illustrating another example of how a scoremanagement node may operate, according to further possible embodiments.

FIG. 4 is a block diagram illustrating a score management node in moredetail, according to further possible embodiments.

FIG. 5 is a block diagram illustrating yet another example of how ascore management node may operate, according to further possibleembodiments.

FIG. 6 is a flow chart illustrating another example of a procedure in ascore management node, according to further possible embodiments.

FIG. 7 is a table with examples of how significance S can be determinedfor different service types, according to further possible embodiments.

FIG. 8 is an example of a significance table sorting five services ofthe highest significance S, according to further possible embodiments.

FIG. 9 is another example of a significance table with differentsub-tables sorting services, device types, access types and Cellidentities of the highest significance S, according to further possibleembodiments.

DETAILED DESCRIPTION

The embodiments described in this disclosure can be used for supportingevaluation of a service by obtaining an estimated user opinion about theservice when it has been delivered to a user by means of atelecommunication network. The embodiments will be described in terms offunctionality in a “score management node”. Although the term scoremanagement node is used here, it could be substituted by the term “scoremanagement system” throughout this disclosure.

Briefly described, a perception score P is calculated that reflects theuser's experience of the service, based on technical networkmeasurements made for one or more events or occasions when the servicewas delivered to the user, hereafter referred to as “service events” forshort. For example, the network measurements may relate to the timeneeded to download data, the time from service request until delivery,call drop rate, data rate and data error rate.

In the following description, any network measurements related todelivery of a service to the user by means of a telecommunicationnetwork are generally denoted “v” regardless of measurement type andmeasuring method. It is assumed that such network measurements v areavailable in the network, e.g. as provided from various sensors, probesand counters at different nodes in the network, which sensors, probesand counters are commonly used for other purposes in telecommunicationnetworks of today, thus being operative to provide the networkmeasurements v to the score management node for use in this solution.Key Performance Indicator, KPI, is a term often used in this field forparameters that in some way indicate network performance.

Further, the term “delivery of a service by means of a telecommunicationnetwork” may be interpreted broadly in the sense that it may also referto any service delivery that can be recorded in the network bymeasurements that somehow reflect the user's experience of the servicedelivery. Some further examples include services provided by operatorpersonal aided by an Operation and Support System, OSS, infrastructure.For example, “Point of sales” staff may be aided by various softwaretools for taking and executing orders from users. These tools may alsobe able to measure KPIs related to performance of the services. Anotherexample is the Customer Care personal in call centers who are aided bysome technical system that registers various user activities. Suchtechnical systems may as well make network measurements related to theseactivities as input to the score management node.

For example, the network measurements v may be sent regularly from thenetwork to the score management node, e.g. in a message using thehyper-text transfer protocol http or the file transfer protocol ftp overan IP (Internet Protocol) network. Otherwise the score management nodemay fetch the measurements v from a measurement storage where thenetwork stores the measurements. In this disclosure, the term networkmeasurement v may also refer to a KPI which is commonly prepared by thenetwork to reflect actual physical measurements. The concept of KPIs iswell-known as such in telecommunication networks.

The perception score P is generated by the score management node asfollows and with reference to FIG. 1 which illustrates a scoremanagement node 100 which receives network measurements v made in atelecommunication network 102. The network measurements v may be sentfrom the network 102 more or less in real-time in a “live stream”fashion, e.g. from an Operation & Maintenance, O&M, node or similar, notshown. Alternatively, the network measurements v may be recorded by thenetwork in a suitable storage or database 104, as indicated by a dashedone-way arrow, which can be accessed by the score management node 100,e.g. at regular intervals, as indicated by a dashed two-way arrow.

The received network measurements v can be seen as “raw data” being usedas input in this procedure. For example, the above O&M node may be anaggregation point or node for distributed sensors and probes that makemeasurements in the traffic flows throughout the network. This node maycombine, correlate and potentially filter the measurement data, e.g. toproduce KPIs or the like.

A quality score Q reflecting the user's perception of quality of adelivered service and an associated significance S reflecting the user'sperception of importance of the delivered service, are determined basedon the network measurements. In this operation, Q and S may bedetermined by applying predefined functions on the network measurements,which will be explained in more detail later below. The perception scoreP is then derived from the quality score Q which is weighted by itsassociated significance S. Basically, the greater significance S thegreater influence has the associated quality score Q on the resultingperception score P.

Before calculating the perception score P, the quality score Q andassociated significance S are also modified in this procedure based on aset of predefined influence factors valid for the user and the deliveredservice. These influence factors may be related to user expectationconsidering various characteristics of the user, correlation ofdifferent service events occurring within a certain time frame, andfading memory of the user which reduces the significance S of a serviceevent over time. The perception score P is then calculated from themodified quality score Q and associated significance S, and theresulting perception score P can then be made available for supportingevaluation of the service. By using this solution, the perception scoreP can be seen as a model for how the user is expected to perceive theservice given the circumstances of the delivered service, which model isbased on objective and technical network measurements.

Returning to FIG. 1, the above-mentioned operation of determining Q andS based on the network measurements is performed by a first scoringmodule 100 a in the score management node 100. Next, the operation ofmodifying Q and S according to the above influence factors is performedby a succession of intermediate scoring modules 100 b, 100 c . . . inthe score management node 100, where each intermediate scoring modulemodifies Q and/or S based on such an influence factor. In this way, thefirst scoring module 100 a determines Q and S purely from the raw data,i.e. the received network measurements, while the intermediate scoringmodules 100 b, 100 c . . . adjust Q and S by considering thecircumstances of the service event which produce the above influencefactors, thereby making Q and S more adapted to the actual situation ofthe delivered service.

Further, the operation of calculating the perception score P from themodified Qm weighted by its associated and modified Sm is performed by aconcluding scoring module 100 x in the score management node 100. Havinggenerated the resulting perception score P, the score management node100 makes P available for evaluation of the service, e.g. by saving itin a suitable storage or sending it to a service evaluation system orcenter, schematically indicated by numeral 106. For example, P may besent to the service evaluation system or storage in an http message oran ftp message over an IP network. The service evaluation system orstorage may comprise an SQL (Structured Query Language) database or anyother suitable type of database.

The quality score Q and associated significance S are thus modifiedgradually in multiple steps by the intermediate scoring modules 100 b,100 c . . . such that the output of modified Q′ and/or S′ from oneintermediate scoring module is used as input to the next successiveintermediate scoring module for further modification, until the thusprocessed data reaches the concluding scoring module 100 x forcalculation of P. It is an advantage that this modular arrangement ofscoring modules 100 a-x in the score management node 100 is flexible inthe sense that any scoring module can easily be added, removed, replacedor modified as desired, without impacting the operation of othermodules.

There are several advantages of this solution as compared toconventional ways of obtaining a user's opinion about a service. First,the perception score P is a quite accurate estimation of the users'opinion of the service event considering the prevailing circumstances,and it is possible to obtain P automatically and continuously inreal-time, basically after every time a service is delivered to a user.There are thus no restrictions regarding the number of users nor theextension of time which makes it possible to obtain a quiterepresentative perception score P. Second, the perception score P iscalculated from technical measurements in the network related to theservice usage which are truthful and “objective” as such, also beingreadily available, thereby avoiding any dependency on the user's memoryand willingness to answer a survey or the like. Third, it is notnecessary to spend time and efforts to distribute surveys and to collectand evaluate responses, which may require at least a certain amount ofmanual work.

Fourth, it is possible to gain further knowledge about the service bydetermining the perception score P selectively, e.g. for specific typesof services, specific types of network measurements, specific users orcategories of users, and so forth. Fifth, it is also possible to trace atechnical issue that may have caused a “bad” experience of a deliveredservice by identifying which measurement(s) have generated a lowperception score P. It can thus be determined when and how a service wasdelivered to a presumably dissatisfied user, as indicated by theperception score P, and therefore a likely technical shortcoming thathas caused the user's dissatisfaction can also be more easilyidentified. Once found, the technical issue can be eliminated orrepaired. Different needs for improvement of services can also beprioritized based on the knowledge obtained by the perception score P.Further features and advantages will be evident in the description ofembodiments that follows.

An example of how the solution may be employed will now be describedwith reference to the flow chart in FIG. 2 which illustrates a procedurewith actions performed by a score management node, to accomplish thefunctionality described above. The score management node is operative tosupporting service evaluation based on a perception score P reflecting auser's experience of a service delivered by means of a telecommunicationnetwork, e.g. in the manner described above for the score managementnode 100.

A first action 200 illustrates that the score management node receivesnetwork measurements related to at least one service event when theservice is delivered to the user. This operation may be performed indifferent ways, e.g. when the network sends a stream of networkmeasurements as they are generated, or by fetching network measurementsfrom a measurement storage, as described above. Action 200 may thus beexecuted continuously or regularly any time during the course of thisprocess of the following actions. The protocol used in thiscommunication may be the hyper-text transfer protocol http or the filetransfer protocol ftp, and the network measurements may be received in amessage such as a regular http message or ftp message. In some possibleembodiments, the score management node may thus receive the networkmeasurements in a message according to the hyper-text transfer protocolhttp or the file transfer protocol ftp.

In some further possible but non-limiting embodiments, the networkmeasurements may be related to any of: the time needed to download data,the time from service request until delivery, call drop rate, data rate,and data error rate. In another possible embodiment, the networkmeasurements may be made during a predefined time interval.

In a next action 202, the score management node determines, in a firstscoring module, a quality score Q reflecting the user's perception ofquality of the delivered service and an associated significance Sreflecting the user's perception of importance of the delivered service,based on the received network measurements. As mentioned above, Q and Smay be determined by applying predefined functions on the networkmeasurements. For example, Q may be determined by applying a firstfunction Q(v) on the network measurements v, and S may be determined byapplying a second function S(v) on the network measurements v. Further,the first and second predefined functions Q(v) and S(v) are dependent ona type of the network measurements used as input to the functions sothat a function applied on, say, measurement of data rate is differentfrom a function applied on measurement of call drop rate, to mention twonon-limiting but illustrative examples.

In a further action 204, the score management node then modifies, in asuccession of intermediate scoring modules, the determined quality scoreQ and associated significance S of each service event based on apredefined influence factor applied in each intermediate scoring module.This means that Q and S that were determined in the first scoring moduleas of action 202, or at least one of Q and S, are modified in a firstintermediate scoring module based on a first predefined influencefactor. The resulting output of the once modified Q′ and S′ is then usedas input to a second intermediate scoring module which modifies Q′ andS′ further based on a second predefined influence factor. The resultingoutput of the twice modified Q″ and S″ may then be used as input to athird intermediate scoring module which modifies Q″ and S″ further basedon a third predefined influence factor, and so forth. The number ofintermediate scoring modules and corresponding influence factors isflexible and can thus be two or more depending on the implementation.

In some possible embodiments which may be used for action 204, thepredefined influence factors may comprise at least two of:

-   -   A) User expectation. In this example, a user profile with        characteristics pertaining to the user is defined and at least        one user group that matches the user profile is identified. The        quality score Q and associated significance S can then be        modified based on predefined group-specific parameters valid for        the at least one identified user group. The group-specific        parameters have thus been defined for a user group to basically        describe the user group. Thus, the user can thereby be described        by means of membership in one or more of these user groups        depending on how relevant the group-specific parameters are to        the user.    -   B) Correlation of different service events. In this example, the        significance S of a quality score Q for a first service event is        modified by multiplying a correlation factor F reflecting a        correlation between the first service event and a second service        event when the first and second service events have both        occurred within a certain time frame. For example, the        correlation factor F may be greater the closer two service        events are in time assuming that if one of the events has        particularly high significance to the user the other event will        also be likely to have high significance to the user if the two        service events occur within a short time frame.    -   C) Fading memory of the user. In this example, the significance        S of each quality score Q is reduced over time according to a        predefined Significance Reduction Rate, SRR assuming that a        user's memory of a service event tends to fade over time and        this can be compensated by reducing the significance of the        service event over time accordingly. By reducing the        significance S over time to simulate the user's fading memory of        the service event, the perception score P will likewise be        reduced over time. The SRR may be defined to form a step-like        function which reduces S in distinct steps over time until it        finally reaches zero assuming that the service event is        virtually forgotten by the user at this point.

After action 204, Q and S have been modified according to the predefinedinfluence factors as exemplified above and the resulting modifiedquality score “Qm” and associated significance “Sm” are used as input inthe next action 206 where the score management node calculates, in aconcluding scoring module, the perception score P based on the modifiedquality score Qm and associated modified significance Sm. Finally, thecalculated perception score P is made available for use in the serviceevaluation, in an action 208, e.g. by sending P to a suitable serviceevaluation system or storage, e.g. as indicated by numeral 106 inFIG. 1. The protocol used in this communication may be e.g. thehyper-text transfer protocol http or the file transfer protocol ftp, andthe perception score P may be sent to the service evaluation system orstorage in an http message or an ftp message over an IP network. Theservice evaluation system or storage may comprise an SQL (StructuredQuery Language) database or any other suitable type of database.

In action 206, the perception score P may be calculated according todifferent possible embodiments as follows. In one possible embodiment,the score management node may calculate the perception score P formultiple service events of service delivery to the user as an average ofmodified quality scores Qm for the service events weighted by theirassociated modified significances Sm. In this case, another possibleembodiment is that the score management node may calculate theperception score P_(N) for N service events of service delivery to theuser according to the following formula:

$P_{N} = \frac{\sum_{n = 1}^{N}{Q_{n}S_{n}}}{\sum_{n = 1}^{N}S_{n}}$where Q_(n) is the modified quality score for a service event n andS_(n) is the associated modified significance for said service event n.In other words, the sum of all N quality scores weighted by theirsignificances is divided by the sum of all the N significances.

It was mentioned above that the network measurements may be made duringa predefined time interval. In another possible embodiment, the scoremanagement node may update the perception score P after a new serviceevent n based on a previous perception score P_(n-1) calculated for aprevious time interval or service event and a quality score Q_(n) andassociated significance S_(n) determined for the new service event n,according to the following formula:

$P_{n} = \frac{{P_{n - 1}S_{{sum},{n - 1}}} + {Q_{n}S_{n}}}{S_{{sum},{n - 1}} + S_{n}}$whereS_(sum,n)=S_(sum,n-1)+S_(n) and P_(n) is the updated perception score.In this way, the perception score P can be kept up-to-date after eachnew service event by using the above simple calculation which adds theinfluence of the new service event n on the total P.

In further possible embodiments, the score management node may identifyat least one type of service for which a modified significance Ssatisfies a threshold condition. If so, the score management node maythen provide the identified at least one type of service as input toroot cause analysis when the perception score P is changedsignificantly. The term “root cause analysis” refers to a procedure fortracing a technical reason for why a service has e.g. been deliveredpoorly, which procedure as such is somewhat outside the scope of thisdisclosure. In this embodiment the root cause analysis is deemed to bewarranted if the perception score P has changed significantly,particularly when P has decreased which indicates that the user isexpected to be dissatisfied with the service as shown by the networkmeasurement(s).

The threshold condition is thus used for finding service events ofunexpected perception score P, either surprisingly low or high. Thisalso makes it easy to exactly identify individual service events thatmay have caused a “bad” experience of a delivered service. For example,the threshold condition may require that the modified significance S ishigh which indicates that the corresponding service event has had agreat influence on the changed P. Thereby, the search for a technicalreason can be focused on that service event to some extent. Someexamples of how such high values of the significance S can be identifiedand maintained as input for the root cause analysis, will be describedin more detail later below with reference to some examples in FIGS. 7-9.

Another example of how the above-described score management node may beconfigured to accomplish the solution is illustrated by the blockdiagram in FIG. 3 which will now be described with further reference toFIG. 2. In this example, a score management node 300 receives variousnetwork measurements v from one or more measurement sources 302 whichmay include an O&M node, a measurement storage, or other suitable entitycapable of supplying such network measurements, examples of which havebeen given above. The network measurements v may be received one by oneas a “live stream”, or multiple measurements may be received at the sametime e.g. at regular intervals. The score management node 300 comprisesfunctionality defined in terms of scoring modules that can be used forimplementing the embodiments described herein. This example involvesfive such scoring modules 300 a-e which effectively form a scoring“pipeline” through which the incoming information is processedsequentially as follows.

In the score management node 300, a first scoring module 300 a, whichmay also be referred to as a “basic” scoring module, determines aquality score Q and an associated significance S for the networkmeasurements, e.g. by applying predefined scoring functions on eachnetwork measurement v being received as raw input data, as of action 202described above. Then each basic pair of Q and S can be seen as a firstversion which is used as input to a succession of intermediate scoringmodules 300 b-d for modification of Q and S based on a predefinedinfluence factor applied in each intermediate scoring module, as ofaction 204.

The initial intermediate scoring module 300 b modifies the basic Q and Sbased on the above-described influence factor A related to userexpectation, thus producing once modified quality score Q′ andsignificance S′. These modified Q′ and S′ are then used as input to thenext intermediate scoring module 300 c which further modifies Q′ and S′based on the above-described influence factor B related to correlationof different service events, thus producing twice modified quality scoreQ″ and significance 5″. These modified Q″ and S″ are then used as inputto the final intermediate scoring module 300 d which further modifies Q″and S″ based on the above-described influence factor C related to fadingmemory, thus producing trice modified quality score Qm and significanceSm. The influence factors A-C have been described above.

Thereby, each pair of quality score Q and associated significance S hasin this example been modified or “adjusted” with consideration to allthe above influence factors A-C, thus making the resulting modifiedvalues of Qm and Sm representative to the user according to the currentcircumstances. Qm and Sm are then used as input to a concluding scoremodule 300 e which calculates the resulting perception score P based onthe modified quality score Qm and associated modified significance Sm.Some examples of how P can be calculated have been described above.Finally, the calculated perception score P is made available for use inthe service evaluation, in this example by storing P in a score storage304 which can be accessed by a suitable entity, not shown, which is usedfor carrying out the service evaluation, e.g. in a business andoperation support system, BSS/OSS. The service evaluation as such issomewhat outside the scope of the embodiments and examples describedherein.

Some examples of how the above-described scoring modules 300 a-e, andalso any further scoring modules described herein, may be implemented inpractice will now be outlined. Each scoring module may be a piece ofsoftware executed by a suitable execution platform. This includes thepossibility to have all scoring modules compiled into one program. Inthis example, the scoring modules may be software modules, e.g. in theform of Java classes, that are compiled together into a single piece ofsoftware that contains the entire score calculation as exemplifiedabove. A scoring coordinator may be used for controlling the operationof each scoring mode, which will be described in more detail later belowwith reference to FIG. 5.

Alternatively, a potentially more flexible implementation may be usedwhere the scoring modules are treated as separate services implementedby distinct pieces of software. They could for example beService-Oriented Architecture, SOA, Web Services. It would also possibleto have the scoring modules implemented as “worker nodes” in a streamprocessing environment such as “Storm”. In general, each scoring moduleis a logical scoring node that can be realized in software and can beeither co-deployed on one physical node or separated and deployed into aset of physical processing nodes.

Since the scoring operation in at least some of the intermediate scoringmodules 300 b-d is dependent on how much time has passed, e.g. after aservice event or between two service events, a virtual clock 300 f maybe employed in the score management node 300 as follows. If the scoringis performed in real-time, or in near real-time, a system clock togetherwith time-stamps that usually come with the raw network measurements canbe used to determine the relevant timing. In case the processing is“offline”, i.e. based on temporarily stored and retrieved networkmeasurements, the time dependent scoring operation needs to be able toreconstruct the timing involved. This is more complex than simplycomparing the difference of time-stamps in the network measurementsbecause the described procedure is a stream-based processing model whereone measurement at a time is scored and older measurements are notpreserved.

This procedure therefore needs to recreate the real time in which themeasured service events occurred, which can be done by the virtual clock300 f recreating “clock ticks” based on timestamps of the receivednetwork measurements. If new measurement data is presented for scoring,the virtual clock 300 f first checks if the timestamp in the data showsa later or earlier time than the time generated by the virtual clock 300f. If the data timestamp is older, the data can proceed in the sequenceof scoring modules.

If the data timestamp is newer by showing a later time than the currentvirtual clock time, the virtual clock 300 f generates clock ticks untilits time becomes later than the timestamp of the data. Clock tickintervals and therefore the resolution of the virtual clock 300 f can beconfigured and the clock tick interval may for example be configured to1 minute. This would mean that all measurement data that has timestampswithin a one minute interval will be processed. If some measurement datais presented that shows the next one minute interval, the virtual clockwill be ticking first to “catch up”, before that data can be processedin the sequence of scoring modules.

It might however happen that several minutes have passed between thetime shown in two consecutive timestamps. Then the clock 300 f willgenerate enough ticks until it has “overtaken” the data time again. Eachclock tick should be generated because the virtual clock 300 f offers a“subscription system” that allows the intermediate scoring modules 300b-d to subscribe for clock notifications about each clock tick interval.In each of the clock tick cycles all these notifications are sent out tothe scoring modules 300 b-d, as indicated by dashed arrows, and therespective operations are finished before the next clock tick isgenerated.

The block diagram in FIG. 4 illustrates another detailed butnon-limiting example of how a score management node 400 may bestructured to bring about the above-described solution and embodimentsthereof. In this figure, the score management node 400 may thus beconfigured to operate according to any of the examples and embodimentsof employing the solution as described above, where appropriate, and asfollows. The score management node 400 in this example is shown in aconfiguration that comprises a processor “Pr”, a memory “M” and acommunication circuit “C” with suitable equipment for receiving andtransmitting data and messages in the manner described herein.

The communication circuit C in the score management node 400 thuscomprises equipment configured for communication with atelecommunication network, not shown, using one or more suitablecommunication protocols depending on implementation. As in the examplesdiscussed above, the score management node 400 is configured or arrangedto perform e.g. the actions of the flow chart illustrated in FIG. 2 inthe manner described above. These actions may be performed by means offunctional units in the processor Pr in the score management node 400 asfollows.

The score management node 400 is arranged to support service evaluationbased on a perception score P reflecting a user's experience of aservice delivered by means of a telecommunication network. The scoremanagement node 400 thus comprises the processor Pr and the memory M,said memory comprising instructions executable by said processor,whereby the score management node 400 is operable as follows.

The score management node 400 is configured to receive networkmeasurements related to at least one service event when the service isdelivered to the user. This receiving operation may be performed by areceiving unit 400 a in the score management node 400, e.g. in themanner described for action 200 above. The score management node 400 isalso configured to determine, in a first scoring module, a quality scoreQ reflecting the user's perception of quality of the delivered serviceand an associated significance S reflecting the user's perception ofimportance of the delivered service, based on the received networkmeasurements. This determining operation may be performed by adetermining unit 400 b in the score management node 400, e.g. in themanner described for action 202 above.

The score management node 400 is further configured to modify, in asuccession of intermediate scoring modules, the determined quality scoreQ and associated significance S of each service event based on apredefined influence factor applied in each intermediate scoring module.This modifying operation may be performed by a modifying unit 400 c inthe score management node 400, e.g. in the manner described for action204 above. The score management node 400 is also configured tocalculate, in a concluding scoring module, the perception score P basedon the modified quality score Qm and associated modified significanceSm, wherein the calculated perception score P is made available for usein the service evaluation. This calculating operation may be performedby a calculating unit 400 d in the score management node 400, e.g. inthe manner described for action 206 above.

It should be noted that FIG. 4 illustrates some possible functionalunits in the score management node 400 and the skilled person is able toimplement these functional units in practice using suitable software andhardware. Thus, the solution is generally not limited to the shownstructure of the score management node 400, and the functional units 400a-e may be configured to operate according to any of the featuresdescribed in this disclosure, where appropriate.

The embodiments and features described herein may thus be implemented ina computer program storage product comprising instructions which, whenexecuted on at least one processor, cause the at least one processor tocarry out the above actions e.g. as described for any of FIGS. 1-6. Someexamples of how the computer program storage product can be realized inpractice are outlined below, and with further reference to FIG. 4.

The processor Pr may comprise a single Central Processing Unit (CPU), orcould comprise two or more processing units. For example, the processorPr may include a general purpose microprocessor, an instruction setprocessor and/or related chips sets and/or a special purposemicroprocessor such as an Application Specific Integrated Circuit(ASIC). The processor Pr may also comprise a storage for cachingpurposes.

The memory M may comprise the above-mentioned computer readable storagemedium or carrier on which the computer program is stored e.g. in theform of computer program modules or the like. For example, the memory Mmay be a flash memory, a Random-Access Memory (RAM), a Read-Only Memory(ROM) or an Electrically Erasable Programmable ROM (EEPROM). The programmodules could in alternative embodiments be distributed on differentcomputer program products in the form of memories within the scoremanagement node 400.

Another example of how the above-described score management node may beconfigured and operable to accomplish the solution, will now bedescribed with reference to the block diagram in FIG. 5 and also to theflow chart in FIG. 6. It was mentioned above that the operation ofscoring modules 300 a-d in FIG. 3 may be controlled by a scoringcoordinator. An example of how this may be done will now be described.FIG. 5 illustrates an example score management node 500 comprising ascoring coordinator 500 a and a series of scoring modules 1-n, denoted500 b-d. In this example, scoring module 500 b represents theabove-described first scoring module 300 a of FIG. 3, scoring module 500c represents the above-described succession of intermediate scoringmodules 300 b-d of FIG. 3, while scoring module 500 d represents theabove-described concluding scoring module 300 e of FIG. 3.

It is assumed that a module registrar 500 e has created a scoring modulesequence for processing network measurements made when a specificservice type is delivered to a user at different service events. Thescoring module sequence thus comprises the scoring modules 500 b-d andit is maintained in a suitable sequence storage 500 f. The scoremanagement node 500 may comprise further scoring modules, not shown, andseveral different scoring module sequences may be maintained in thesequence storage 500 f for different service types. Each networkmeasurement v is first received by the scoring coordinator 500 a, as ofaction 600. In a next action 602, the scoring coordinator 500 a readsthe scoring module sequence from the storage 500 f and sends the networkmeasurement v to the first scoring module 500 b of the scoring modulesequence, in a following action 604.

When the first scoring module 500 b has determined Q and S from thenetwork measurement v, e.g. as of action 202, the scoring coordinator500 a receives Q and S as scoring data D from the first scoring module500 b, in an action 606. In a next action 608, the scoring coordinator500 a again reads the scoring module sequence from the storage 500 f tofind the next scoring module. The scoring coordinator 500 a thusdetermines if there is any further scoring module in the scoring modulesequence, in an action 610. In this case, an intermediate scoring module500 c is found to be next in the sequence and the scoring coordinator500 a accordingly sends the scoring data D to that scoring module 500 cfor modification, in a following action 612, and scoring module 500 cthen modifies the scoring data D, e.g. as of action 204. In anotheraction 614, the scoring coordinator 500 a receives the modified scoringdata D′ from the intermediate scoring module 500 c.

The scoring coordinator 500 a now repeats the procedure by returning toaction 608 to read the scoring module sequence again from the storage500 f. If it is then determined in action 610 that there is a furtherscoring module in the sequence, the scoring coordinator 500 aaccordingly sends the scoring data D′ to the next scoring module inaction 612 for further modification and receives modified scoring datain action 614.

After repeating actions 608-614 a number of times, all scoring modulesin the score management node 500 have been identified and used, the lastscoring module being the concluding scoring module 500 d whichcalculates the resulting perception score P, e.g. as of action 206, andthe scoring coordinator 500 a eventually determines in action 610 thatthere is no further scoring module in the scoring module sequence. Theprocedure then ends when the final result, i.e. the calculatedperception score P, is stored in a suitable service evaluation system orstorage 502, as shown in a final action 616, e.g. as of action 208. Forexample, the concluding scoring module 500 d may itself store Pinstorage 502, as indicated in FIG. 5, or P may be delivered to thescoring coordinator 500 a which in turn stores P.

As described above, the concluding scoring module calculates theperception score P based on the modified quality score Qm and associatedmodified significance Sm, which may be done for multiple service eventsof service delivery to the user as an average of modified quality scoresQm for the service events weighted by their associated modifiedsignificances Sm. In action 616, the concluding scoring module 500 d maythus update an already stored value of P after each new service eventhaving generated a network measurement that is processed according toFIG. 6.

Each scoring module 1-n thus processes input information and in theseoperations they use and apply certain operational parameters and/orformulas, e.g. as described above for respective modules. Theoperational parameters and/or formulas are denoted “module parameters”for short which may be maintained in a parameter storage 500 g that isaccessed by the scoring modules 500 b-d when executing their respectivescoring operations.

It was mentioned above that the score management node may identify atleast one type of service for which a modified significance S satisfiesa threshold condition, and that the identified at least one type ofservice may then be provided as input to root cause analysis when theperception score P is changed significantly. Examples of how this can bedone will now be described. It is assumed that the resulting modifiedsignificance S can be detected and collected, e.g. the output from thelast intermediate scoring module 100 c or 300 d being the modifiedsignificance Sm, in order to generate a table with services that havegenerated the highest significances as follows.

The final modified significance S of a single service event may thus beused in order to determine what type of service did get the highestoverall significance. In this case the significances determined for acertain service type are summed up and the sum value is stored. In thisway, a significance table can be built that shows which types ofservices did have the highest significance in the calculation of theperception score. The significance table can be sorted according to thesignificance sums resulting in a list with the most significant serviceevent on top of the list. This shows what type of service has producedthe highest weight in the calculation of the perception score P.

An example of such a significance table is shown in FIG. 7 with entriesfor different service types and their resulting significance sum, thenumber of scorings of service events and a calculated average of thesignificance for all service events. Whenever a new scoring for aservice type Tx with a significance S is obtained, S is added to thesignificance sum S_Tx of the service type Tx. In this table, also thenumber of scorings and the average significance are kept for eachservice type. This provides further information indicating whether thesignificance of a service type is coming from a small number of verysignificant service events or from a large number of less significantones. This may provide further insights into the service event historyof the user and the root cause for the perception score.

A table like this is associated with the perception score P. Thus forevery perception score P, a table of the most significant experienceevents can be made available. As similar to the perception score P, thistable is user specific and this kind of table can be generated for eachuser.

It may be of interest to find out why the perception score P hasincreased or declined, and this significance table can indicate whattypes of services had the greatest influence on changes in theperception score. Further investigations in the root cause analysis canthen focus on these service types accordingly.

Returning to FIG. 3, a significance table generator 300 f is illustratedwhich extracts the modified significance Sm and other relatedinformation from the output of the final intermediate scoring module 300d. The significance table generator 300 f then generates a significancetable 306, e.g. similar to the table of FIG. 7, which can be accessed bya service evaluation system in order to perform the root cause analysisbased on the information in the significance table 306. The significancetable may be reset regularly, e.g. once a day, and then re-built fromscratch. Thereby, only recent significances will occur in thesignificance table. If it is desirable to investigate longer timeperiods, each generated single day table may be stored and a “multi-day”table can easily be calculated by summing up the entries from multiplesingle day tables. Manual root cause analysis time and therefore costscan be saved because this arrangement of significance table(s) allows toget automatic indications of where to focus the root cause analysis.

The significance table may require a minimum significance threshold.This would only allow service events with high significance exceedingthe minimum significance threshold to be in the significance table. In apossible embodiment, the above-mentioned threshold condition may thusdictate that the modified significance S is above a predefined thresholdvalue. In this case the number of entries in the significance table mayvary depending on how many service events fulfil the thresholdcondition.

In another possible embodiment, the threshold condition mayalternatively dictate that the modified significance S is above a lowestvalue of modified significances S for a set of previously identifiedservice types. FIG. 8 is a significance table illustrating an example ofhow this embodiment may be employed. Generally, a significance table ofthe n most significant single service events may be maintained withcertain related information. n=10 would mean that the 10 mostsignificant events are maintained in the significance table. If a newservice event is scored that gets a higher significance S than the leastsignificant service event in the significance table, the leastsignificant service event is deleted and the new one is added to thesignificance table. Further, this significance table of the n mostsignificant service events can be archived and reset regularly.

It is thus possible to generate a table of individual significantservice events comprising detailed information related to the n mostsignificant service events. If n is 10, this means the 10 mostsignificant events are kept with related information including servicetype, significance S, time of the event, and a parameter called KeyPerformance Indicator, KPI, which is a term often used for parametersthat in some way indicate network performance. The KPI thus correspondsto the above-described network measurements v. If a service event isscored that is more significant than the least significant one in thetable, the data of the new service event is added to the table.

The relevant event information in this table may also include atimestamp of the service event and the related KPI and measurements.Details related to the scoring procedure may also be preserved, e.g.including the influence factors and parameters applied by the scoringmodules to modify the significance. This allows detailed understandingand improvement of the scoring process.

The significance table in FIG. 8 comprises information about n=5 mostsignificant service events. If a service event with significance greaterthan 9 is scored, the last service event in the table concerning a videosession at time 11:10 will be deleted and the new service event will beadded to the table. Also, this table can be reset regularly while theold values are moved to a suitable table storage or archive.

In further possible embodiments, the score management node may identifythe at least one type of service based on an average significancedetermined for one or more of: a certain access technology, a certaindevice type or manufacturer, and a certain cell identity. Theseembodiments are exemplified by the significance table in FIG. 9. In thistable, the significance is not exclusively tied to individual serviceevents. Further criteria are used here in order to calculatesignificance sums. For example, the raw network measurements mightcontain information about the user's device type and devicemanufacturer. Also, the access type for wireless communication, e.g. 2G,3G, LTE or Wifi, or a cell identity might be known and used. Thesefactors may provide further dimensions to maintain significance sums.

The significance table in FIG. 9 illustrates an example of how some ofthe above factors may be used as criteria for generating thesignificance table. In this example, the significance information isrelated to device type used by the user, and also features of thenetwork including access type used for the service events and the cellwhere the service events took place. Each single service event and itssignificance are added once to each of the sub-tables. The creation ofthe tables exemplified by FIGS. 7-9 may be hard coded or simple rulesmay be used that connect a reported property like a cell ID to a newtable.

While the solution has been described with reference to specificexemplifying embodiments, the description is generally only intended toillustrate the inventive concept and should not be taken as limiting thescope of the solution. For example, the terms “score management node”,“scoring module”, “perception score”, “quality score”, “significance”,“service event”, “threshold condition” and “significance table” havebeen used throughout this disclosure, although any other correspondingentities, functions, and/or parameters could also be used having thefeatures and characteristics described here. The solution is defined bythe appended claims.

The invention claimed is:
 1. A method performed by a score managementnode, comprising a processor and a memory containing instructionsexecutable by the processor, for supporting service evaluation byobtaining a perception score P reflecting a user's experience of aservice delivered by means of a telecommunication network, the methodcomprising the steps performed by the score management node: receivingnetwork measurements related to at least one service event when theservice is delivered to the user, determining, in a first scoringmodule, a quality score Q reflecting the user's perception of quality ofthe delivered service and an associated significance S reflecting theuser's perception of importance of the delivered service, based on thereceived network measurements, modifying, in a succession ofintermediate scoring modules, the determined quality score Q andassociated significance S of each service event based on a predefinedinfluence factor applied in each intermediate scoring module, andcalculating, in a concluding scoring module, the perception score Pbased on a modified quality score Qm and an associated modifiedsignificance Sm, wherein the calculated perception score P is madeavailable for use in the service evaluation, wherein the scoremanagement node calculates a perception score P_(N) for N service eventsof service delivery to the user as$P_{N} = \frac{\sum_{n = 1}^{N}{Q_{n}S_{n}}}{\sum_{n = 1}^{N}S_{n}}$where Q_(n) is a modified quality score for the service event n andS_(n) is an associated modified significance for the service event n. 2.The method according to claim 1, wherein the score management nodecalculates the perception score P for multiple service events of servicedelivery to the user as an average of modified quality scores Qm for theevents weighted by their associated modified significances Sm.
 3. Themethod according to claim 1, wherein the network measurements arerelated to any of: a time needed to download data, a time from servicerequest until delivery, a call drop rate, a data rate, and a data errorrate.
 4. The method according to claim 1, wherein the networkmeasurements are made during a predefined time interval.
 5. The methodaccording to claim 4, wherein the score management node updates theperception score P after a new service event n based on a previousperception score P_(n-1) calculated for a previous time interval orservice event and the quality score Q_(n) and the associatedsignificance S_(n) determined for the new service event n, as$P_{n} = \frac{{P_{n - 1}S_{{sum},{n - 1}}} + {Q_{n}S_{n}}}{S_{{sum},{n - 1}} + S_{n}}$where S_(sum,n)=S_(sum,n-1)+S_(n) and P_(n) is an updated perceptionscore.
 6. The method according to claim 1, wherein the predefinedinfluence factor comprise at least two of: an user expectation wherein auser profile with characteristics pertaining to the user is defined andat least one user group that matches the user profile is identified, andwherein the quality score Q and associated significance S are modifiedbased on group-specific parameters valid for the at least one identifieduser group, a correlation of different service events wherein thesignificance S of a quality score Q for a first service event ismodified by multiplying a correlation factor F reflecting a correlationbetween the first service event and a second service event when thefirst and second service events have both occurred within a certain timeframe, and a fading memory of the user wherein the significance S ofeach quality score Q is reduced over time according to a predefinedsignificance reduction rate, SRR.
 7. The method according to claim 1,wherein the score management node identifies at least one type ofservice for which the modified significance S_(m) satisfies a thresholdcondition, and provides the identified at least one type of service asinput to a root cause analysis when the perception score P is changedsignificantly.
 8. The method according to claim 7, wherein the thresholdcondition dictates that the modified significance S_(m) is above apredefined threshold value.
 9. The method according to claim 7, whereinthe threshold condition dictates that the modified significance S_(m) isabove a lowest value of modified significances S_(m) for a set ofpreviously identified service types.
 10. The method according to claim7, wherein the score management node identifies the at least one type ofservice based on an average significance determined for one or more of:a certain access technology, a certain device type or manufacturer, anda certain cell identity.
 11. The method according to claim 1, whereinthe score management node receives the network measurements in a messageaccording to the hyper-text transfer protocol http or the file transferprotocol ftp.
 12. A score management node arranged to support serviceevaluation by obtaining a perception score P reflecting a user'sexperience of a service delivered by means of a telecommunicationnetwork, the score management node comprising a processor and a memorycontaining instructions executable by the processor, whereby the scoremanagement node is configured to: receive network measurements relatedto at least one service event when the service is delivered to the user,determine, in a first scoring module, a quality score Q reflecting theuser's perception of quality of the delivered service and an associatedsignificance S reflecting the user's perception of importance of thedelivered service, based on the received network measurements, modify,in a succession of intermediate scoring modules, the determined qualityscore Q and associated significance S of each service event based on apredefined influence factor applied in each intermediate scoring module,and calculate, in a concluding scoring module, the perception score Pbased on a modified quality score Qm and an associated modifiedsignificance Sm, wherein the calculated perception score P is madeavailable for use in the service evaluation, wherein the scoremanagement node calculates a perception score P_(N) for N service eventsof service delivery to the user as$P_{N} = \frac{\sum_{n = 1}^{N}{Q_{n}S_{n}}}{\sum_{n = 1}^{N}S_{n}}$where Q_(n) is a modified quality score for the service event n andS_(n) is an associated modified significance for the service event n.13. The score management node according to claim 12, wherein the scoremanagement node is configured to calculate the perception score P formultiple service events of service delivery to the user as an average ofmodified quality scores Qm for the events weighted by their associatedmodified significances Sm.
 14. The score management node according toclaim 12, wherein the network measurements are related to any of: a timeneeded to download data, a time from service request until delivery, acall drop rate, a data rate, and a data error rate.
 15. The scoremanagement node according to claim 12, wherein the network measurementsare made during a predefined time interval.
 16. The score managementnode according to claim 15, wherein the score management node isconfigured to update the perception score P after a new service event nbased on a previous perception score P_(n-1) calculated for a previoustime interval or service event and the quality score Q_(n) and theassociated significance S_(n) determined for the new service event n, as$P_{n} = \frac{{P_{n - 1}S_{{sum},{n - 1}}} + {Q_{n}S_{n}}}{S_{{sum},{n - 1}} + S_{n}}$where S_(sum,n)=S_(sum,n-1)+S_(n) and P_(n) is an updated perceptionscore.
 17. The score management node according to claim 12, wherein thepredefined influence factor comprise at least two of: an userexpectation wherein a user profile with characteristics pertaining tothe user is defined and at least one user group that matches the userprofile is identified, and wherein the quality score Q and associatedsignificance S are modified based on group-specific parameters valid forthe at least one identified user group, a correlation of differentservice events wherein the significance S of a quality score Q for afirst service event is modified by multiplying a correlation factor Freflecting a correlation between the first service event and a secondservice event when the first and second service events have bothoccurred within a certain time frame, and a fading memory of the userwherein the significance S of each quality score Q is reduced over timeaccording to a predefined significance reduction rate, SRR.
 18. Thescore management node according to claim 12, wherein the scoremanagement node is configured to identify at least one type of servicefor which the modified significance S_(m) satisfies a thresholdcondition, and to provide the identified at least one type of service asinput to a root cause analysis when the perception score P is changedsignificantly.
 19. The score management node according to claim 18,wherein the threshold condition dictates that the modified significanceS_(m) is above a predefined threshold value.
 20. The score managementnode according to claim 18, wherein the threshold condition dictatesthat the modified significance S_(m) is above a lowest value of modifiedsignificances S_(m) for a set of previously identified service types.21. The score management node according to claim 18, wherein the scoremanagement node is configured to identify the at least one type ofservice based on an average significance determined for one or more of:a certain access technology, a certain device type or manufacturer, anda certain cell identity.
 22. The score management node according toclaim 12, wherein the score management node is configured to receive thenetwork measurements in a message according to the hyper-text transferprotocol http or the file transfer protocol ftp.
 23. A computer programstorage product comprising instructions which, when executed on at leastone processor, cause the at least one processor to carry out the methodaccording to claim 1.